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ABSTRACT 

The goal of the dependability analysis is to predict inconsistencies and to reveal ambiguities and incompleteness in the 
designed learning scenario. Evaluation, in traditional learning design, is generally planned after the execution of the 
scenario. In mobile learning, this stage becomes too difficult and expensive to apply due to the complexity and 
heterogeneity of mobile infrastructure with contextual constraints. In other word, the test of the dependability has to be 
alone at early stage. To achieve our goal, evaluating the dependability of the learning scenario, the evaluation model is 
constructed from the analysis of literature by exploring different aspects of dependability. We propose to apply formal 
specification and verification in order to evaluate the functional dependability of the mobile learning environment. A 
specification is the description (usually by means of a temporal logic formulas) of the property (or temporal behaviour) to 
be fulfilled by the environment under consideration. Verification consists on (exhaustively) proving that the learning 
environment is correct. 
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1. INTRODUCTION 

Dependability is an important pre-requisite for mobile learning (Magal-Royo, T. et al, 2007) and should be 
evaluated sufficiently and at early stage to respond user’s requirement. In fact, the learning environment had 
to operate as learners and designers expect and that it will not fail in normal use (Le, Q. and Le, T, 2007). 

Many researchers have defined the dependability in slightly different way. Dependability is an orthogonal 
issue that depends on QoS. We consider its original meaning as defined in (Laprie, J.-C, 1985), (Avizienis, A. 
et al. 2004) : “ Dependability is the quality of the delivered such that reliance can justifiably be placed on this 
service It could be typically divided into a number of aspects namely (Avizienis, A. et al. 2004): 
(functional) correctness, safety, security, reliability, availability, transparency and traceability. Each 
dimension includes analysis techniques, assessment methods and measures. Mobile learning environments 
come with requirements from all aspects of dependability. The reason for this lies in the nature of these 
mobile scenarios itself. Typically these environments are very tightly connected with specific users. 

Dependability evaluation is an important activity as well as evaluation of the executed learning scenario. 
Post-evaluation techniques and approaches take much time and cost, and an early preventive evaluation could 
be one of the key factors to cost effective mobile learning development. 

The basic question is: How to evaluate the learning functionalities in their context of use at early stage in 
order to ensure the dependability of the conceived scenario ? 

To answer to this question, we first summarize our definitions. We define learning scenario as 
technological environment consisting of one or more activities, correlated together offering a complete 
scenario of information and communication services required for supporting learning. A mobile learning can 
be defined as “... any service or facility that supplies a learner with general electronic information and 
educational content that aids in acquisition of knowledge regardless location and time...” (Lehner, F. and 
Nosekabel, H. 2002.). 
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This paper proposes an early preventive evaluation framework: ReStart-Me (Re-engineering educational 
Scenario based on timed automata and tracks treatment for Malleable learning environment) that intends to 
reduce the time and cost through using test cases as a means of the evaluation. Test cases are developed in the 
process of the leaning scenario design and used to test the target scenario. ReStart-Me checks formally 
inclusion relation between dependability requirements and test cases. 


2. BACKGROUND AND RELEATED WORKS 
2.1 What can we evaluate in Mobile Learning Scenario? 

As computers have become more rapid and powerful, educational software has flourished and there are 
numerous claims conceived by designers. Thus evaluation software is important so that teachers can make an 
appropriate choice of learning scenarios and which are suitable to the teaching and learning context. 

According to (Elissavet, G. and Economides, A. A. 2003) we can evaluate these following dimensions in a 
learning scenario: content, technical support, learning process (see figure 1). Each dimension includes a 
number of sub categories and criteria that could be considered in the evaluation process. All these aspects are 
equally important, as the learning activity has to be simultaneously pedagogically and technically sound. 

lTowever, we propose that the learning activity’s context should also be considered for evaluation. The 
context is a set of evolutive elements appropriate to the interaction between learner and learning application 
including the learner and the learning environment themselves. Figure 1 presents in a diagram the different 
aspects that we can combine and include in the evaluation framework. 

Mobile learning could be evaluated from different viewpoints: The first one is a technical oriented 
perspective. The content suitable for m-learning needs to be available and adequate to the learning 
enviromnent. The second one is pedagogical oriented perspective. It points out that m-learning develops new 
skills and approaches to ensure the pedagogical effectiveness. We consider that the context of mobile 
enviromnent should be also considered in evaluation process. The features of activity’s context are: location, 
network, user, time and device. 

The device dimension: we consider that it is important capabilities of user’s device, especially 
hardware attributes, for mobile learning due to the fact they have a big impact on learning scenario 
execution. 

The network and connectivity dimension: nowadays mobile device might be connected to the 
“Net” via many technologies: GPRS, UMTS, WiFi, 3G-telecommunication, etc. Mobile devices 
often have periods of disconnection that had to be considered in evaluation of the dependability of 
the learning scenario. The connectivity quality depends on user’s location and mobility. 



Figure 1. Different features that could be evaluate in the learning scenario 


2.2 Evaluation of Mobile Learning Scenarios: Related Works and Issues 

There have been several researches to develop methods to evaluate mobile learning scenario. The most 
widely used methods are: 
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Heuristic evaluation method based on principals and many categorised dimensions (Nielsen, J. and 
Molich, R. 1990). A wide range of methods has been developed to systematically evaluate the 
quality of information technologies. 

Evaluation methods based on Simulations (Poison, L. C. et al. 1990) where some user problems are 
simulated in details, especially analysing each task from a cognitive point of view. 

Evaluation system based on tracks analysis requests both analytical and technical staff. The first 
group is responsible for defining scores for various features of the e learning scenario according to a 
specific set of evaluation coefficients. This team also specify the quality of the learning scenario. 

The technical staff develops the system or specify the evaluation framework to the mentioned scores 
(Zorrilla, M. and Alvarez, E. 2008) (Ben Sassi, M. and Laroussi, M. 2012). 

Teaching test methods are based on an appropriate testing program that is suitable to different goals 
and characteristics of the teaching style. These methods implement the pre-test, the post-test of the 
learning scenario, and other steps to test the knowledge of students and their skills and finally 
determine the effectiveness of teaching scenario (JIAa, Z. and Han, X. 2013). 

As shown, there are several evaluation studies. However, these methods vary widely in their evaluation 
scope, outcomes and techniques. In fact, in the one hand, there are generic methods that, although useful in 
theory, are not very applicable in practice, since they do not take into account the situatedness of the 
courseware evaluation, determined by the context of the learning scenario. In the other hand, some of these 
methods described above have a very specific target (cognitive overview). Nevertheless, with the growth in 
the use of mobile technologies and the learning scenario database, an increasing number of teachers want to 
reuse their scenario in different contexts. 

Instead of adding to the already large number of checklists for learning scenario evaluation, we are 
attempting, in the following section, to address contextual evaluation based on formal modeling and 
verification of educational scenario. Next, the evaluation framework is discussed. 


3. RESTART-ME: A FORMAL EVALUATION FRAMEWORK FOR M- 
LEARNING 

3.1 Restart-Me Challenges 

In order to fill the gap between learning scenarios evaluation methods and mobile learning scenarios, we 
present in this paper, a formal method to evaluate scenarios at an early stage: ReStart-Me (Re-engineering 
educational Scenario based on timed automata and tracks treatment for Malleable learning environment). 
This formal evaluation is based on automata theory and formal verification and aims to: 

Avoid costly and time-consuming scenarios implementation or deployment 

Simulate scenario execution on real time in order to check properties and to detect errors (such as 
deadlocks and liveness) and then to regulate scenario with timing constraints. 

The evaluation framework is based on different kind of context: Actors, Location, Device, Network and 
connectivity. These elements are motivated by an empirical analysis of different definitions of learning 
context. In fact, these elements represent core features of any educational scenario supporting any learning 
approaches such as hybrid learning, mobile learning, ubiquitous learning, pervasive learning,... Consequently, 
they represent interesting evaluation criteria in order to capture at early stage inconsistencies and design 
weakness. 

3.2 Conceptual Framework 

Figure 2 overviews the proposed software dependability evaluation framework for mobile learning. ReStart- 
Me intends to reduce the cost of learning environment quality evaluation through using test cases. It uses 
formal checking techniques to ascertain inclusion relation between dependability requirements and test cases. 
Dependability requirements and test cases are modeled into timed automata. Then, they are transformed into 
temporal logical properties to formal checking methods. If the formal checking produces TRUE then we can 
conclude that the dependability properties are well implemented into the learning environment. On the other 
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hand, in case of FALSE, we generate an errors report and recommendations to assist designers to ensure 
learning scenario quality. 

Learning Scenario Evaluation Framework ' ' 




Dependability properties Models of test cases 

g a 

/ v 

Dependability properties i Dependability properties | 

are well implemented J are not implemented J 


Figure 2. Dependability Evaluation Framework for Learning Scenario. 

The evaluation process contains essentially these three steps (see Figure 2) inspired from (Coronato, A. 
and De Pietro, G. 2010): 

- Pre Step 0: Behavioural Modelling: this phase provides informal descriptions of both the learning 
environment and the correctness properties to be checked. Pedagogues and designers have to 
participate in this phase. 

- Step 1: Formal Modeling of Educational Scenario: In this step, conceptual designers identify the 
entities to be considered. Designers have to successively refine it and decide which the final actors 
to be modelled are. The content of each process is created and redefined by modelling the dynamic 
behaviour. Then, we extend the obtained automaton with global and local clocks. We also define 
contextual constraints and correlation between different activities. By creating a formal specification 
of the learning scenario, the designers are forced to make and to define a detailed scenario analysis 
at early stage before its deployment into the learning system. 

- Step 2: Tracks Simulation: Through simulation, we can observe all possible interactions between 
automata corresponding to different entities involved in the learning scenario. This step generates 
simulated tracks that facilitate errors detection. A simulation is equivalent to an execution of the 
designed learning scenario. It gives some insight on how the model created behaves. 

- Step 3: Properties verification : with formal verification, we check the correctness of the designed 
learning scenario. This process aims to build a remedial scenario and to help designer in the 
reengineering process by providing Errors and warning reports. 

ReStart-Me can be applied with the help of different developing methods and tools. To exemplify this, we 
have selected a tool that is well known, efficient and provide us with most of what we need. The selected tool 
is a model checker that has a modelling language, plus simulation and formal verification capabilities. We 
think that our choice is well founded but is not essential. To illustrate the application of ReStart-Me, we will 
use UPPAAL model checker (Bengtsson, J. et al. 1996). 


4. EXPERIMENT AND RESULTS 

The objective of this study reported in this section is twofold: (i) to check practicality of the proposed 
framework when building an actual mobile scenario (ii) and to evaluate qualitatively the benefits of such 
methodology. A team of researchers composed mainly by of five Designers has prepared a textual description 
of the malleable learning scenario. 

These meetings have produced a scenario design document that describes the sequencing of different 
activities and rules of the malleable learning scenario. To summarize, the learning scenario can be described 
as follows: 
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A high school decides to raise awareness of pupils on the effects of pollution on the environment by 
organizing a trial allowing the follow-up of the pupil’s educational curriculum in the field of “Education 
about the environment”. This trial enables pupils to learn through factual cases and to experiment various 
scenarios using pervasive and mobile technologies. 

The physical setting of this trial, where activities take place, is an ecological zone near by an industrial 
area. 

In order to boost intra-group competition, students were divided in three groups under the supervision of 
their coach and each group consisted of six pupils. Additionally, each group was divided in two subgroups of 
three students. The ultimate goal behind this clustering is to reinforce teamwork and collaboration within the 
individual subgroups and to make it a collaborative and challenging game that takes place in different 
locations. 

The outdoor subgroup is equipped with a smart phone with a wireless connection. At the beginning of the 
first stage, a localization sensor localizes the out door’s subgroup and a notification is sent to ask students to 
identify and take a photo of the QR-code stuck to a tree. Instantly, a text adapted to the pupils’ level and 
pictures that visualize and describe the activities to accomplish in the current stage is displayed on the screen 
of the smart phone. 

In the first stage, the outdoor subgroup can take a photo of a plant and search for a plant related groups 
and then share the photo and ask for help in identifying it. After a pre-defined time, the subgroup will receive 
a stage-adapted quiz via automatic text message. Pupils need to write an answer using their smartphone and 
submit it. If the answer submitted by the group is not correct, the system sends an alert to the coach 
informing him/her that pupils need some support. The coach should send to them some hints. 

In order to improve the coaching task, tutor decides that after three wrong attempts, the pupil is guided to 
start learning session by using his mobile device. The e-learning client allow the student to directly mash up 
widgets to create lesson structure and add powerful online test widgets, communication widget (chat, forum 
and personal messages), content scheduling widgets, communication tracking, announcements, content 
flows, cooperative content building widgets. 

The student could drag from the widget repository and drops into the elearning client UI all the widgets 
needed for providing video, audio and other multimedia content. The session of learning take 30 minutes 
with GPRS connection and 40 minutes with EDGE connection. 

Else, if the answer is correct the indoor subgroups will receive the list of activities of the second stage and 
will get joined by their corresponding outdoor subgroups that will hand over the picked plant samples. 

At the end, indoor and outdoor subgroups of each of the three groups should collaborate in drawing their 
own conclusions using the collaborative tool Google Docs and then present the outcome of their study about 
effects of pollution on environment. 


Table 1. Context Information (Network connection quality) 



Zone 1 

Zone 2 

Type of network 

EDGE 

GPRS 


4.1 Modelisation and Evaluation 

Figures 3 and 4 provide an overview of different automata modeled for the planned learning activities 
(outdoor activities, quiz, and lesson) and corresponding to student. We define a global variable “clock” 
named Time that gives idea about the duration of each activity. The timing constraints associated with 
locations are invariants. It gives a bound on how long these locations can be active. We also define other 
global variable Power to calculate the energy of battery of the smartphone. 

In order to facilitate the learning scenario analysis, we model each activity separately. The idea is to 
define templates for activities that are instantiated to have a simulation of the whole scenario. The motivation 
for the use of templates is that the understanding, the share and the reuse of different components of the 
learning scenario become easier. 
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Figure 3. The automata model of different activities for one actor (Student) 



Figure 4. The automaton model of the smart phone 

The whole scenario is modelled as a parallel composition of timed automata. An automaton may perform 
a transition separately or synchronise with another automaton (channel synchronisation) or it can be activated 
after a period of time through flags. 

Based on automata presented above, tracks simulations are generated visualizing all possible interactions 
between different actors. A screen dump of the simulation of the designed educational scenario is below (see 
figure 5). 

In order to help designers to improve their educational scenario and to obtain better outcomes, through 
the generated simulations, we try to localize design errors, to answer and to verify the following questions: 

Does the description of the learning scenario clearly define time constraints for each activity 
and the whole scenario? 

Is there any situation of deadlocks, liveness or starvation? 

A possible situation of deadlock is detected within tracks simulation of the whole scenario; In fact, 
students could have a period of inactivity especially when they begin their lesson in the first zone (EDGE) 
where the network connectivity is lower and smartphone’ energy is limited. To check this property, we are 
based on reachability property that is considered as the simplest form of properties. They ask whether a given 
state formula, possibly can be satisfied by any reachable state. Another way of stating this is: Does there exist 
a path starting at the initial state, such that “the battery state” is eventually over along that path? 
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We traduce this property in temporal logic formula: “ EoSmartphone.BatteryjOver ” and this property is 
verified as shows figure 6. Then, we conclude that dependability is not assured and we had to adapt mobile 
application to the learning environment context (if the student is located in Zone 1, he had to reduce the 
energy consumption by loading the necessary widgets). 



Figure 5. Simulation of the modeled Learning scenario 
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Figure 6. Checking the reachability of the battery state 


4.2 Result 

The final evaluation was conducted using interviews with five pedagogical designers. During the interview, 
the engineers have addressed several points about there experiences with the presented methodology. 
Hereafter a summary of the main points that have been discussed: 

- Fours engineers of five have found the methodology very useful in identifying the inconsistencies 
functional aspects of the learning scenario. In fact the culture of classical learning scenario 
evaluation does not highlight the importance of these functional aspects (like time, context of 
learning, ...). In the one hand, thanks to the formal modelisation, we can simulate and check all 
possible execution exhaustively. In the other hand, it allows us to redefine the conceived learning 
scenario in more details deeply at early stage in order to avoid the problems highlighted by the 
evaluation (or on the contrary, to rethink the first design). 

- Two engineers have already some experience in developing large learning scenario. So they were 
aware of the cost of their implementation and revision. Consequently, they pointed out that the early 
evaluation stage performed before implementation and deployment has helped to revise some 
fundamental decision without having to conduct costly implementation. In fact, the contextual and 
formal evaluation shows us if the resulting evaluation is exactly what we expect from the 
modelisation of learning scenario (and from the learner) before the implementation stage. It may 
happen, for example, that we want to create a correctly and general design for a complex scenario 
but the result of the formal evaluation shows that we lack in contextual constraints definition and 
specification. 
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- Three engineers of five have pointed out that the formal modeling and verification is not so easy to 
build. Indeed, this formalism is not very well known by all conceptual designers. For this reason, we 
are developing a formal modelisation assistant tool to help no-skilled designers to evaluate the 
conceived learning scenario. 


5. CONCLUSION 

In this paper, we presented different dimensions that could be evaluated in learning scenario. We proposed a 
formal framework evaluation to support a contextual evaluation in dependable learning environment. This 
framework based on formal modelisation and verification allows to designers to simulate the conceived 
learning scenario with contextual constraints. 

Complex content management is employed in m-learning, when there is an interaction between teachers 
and students. The limit is established by the available technology. It is still a determinant factor in such 
technologies, the transmission cost and speed, which affects the communication logistic between teacher and 
student and has a direct repercussion on the learning type. For this reason, we have attempted to demonstrate 
the benefits of the proposed approach by presenting a realistic case study. We think that this formal 
methodology of evaluation provides useful information for the re-use and the reengineering of learning 
materials. In the one hand, it can solve many difficulties in getting proper information about the students and 
their behaviour (ethical problems, track s' collect,...). 

In the other hand, the use of a rigorous formalism to describe a model of deployment allows a better and 
more precise understanding of the planned scenario. It provides the designer with an analytical model that 
helps him to detect errors and learner's difficulties through test case generation and deductive verification. 
Finally, it creates an environment that simulates an external reality and learner's behaviour. 

For going farther this first result, we are currently working on two directions: Firstly, we will attempt to 
deepen our proposal and to apply formal verification relying on temporal logical formulas and model 
checking engine. Secondly, in order to assist the designer in expressing these properties, we are developing a 
tool that helps him/her to draw this business requirement and to generate error report automatically. 
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